<!doctypehtml><html class="sidebar-visible no-js light"lang=en><head><meta charset=UTF-8><title>Dealing with duplicates - Perl One-Liners Guide</title><meta content="text/html; charset=utf-8"http-equiv=Content-Type><meta content="Example based guide for text processing with Perl from the command line"name=description><meta content=width=device-width,initial-scale=1 name=viewport><meta content=#ffffff name=theme-color><meta content="Perl One-Liners Guide"property=og:title><meta content=website property=og:type><meta content="Example based guide for text processing with Perl from the command line"property=og:description><meta content=https://learnbyexample.github.io/learn_perl_oneliners/ property=og:url><meta content=https://raw.githubusercontent.com/learnbyexample/learn_perl_oneliners/main/images/perl_oneliners_ls.png property=og:image><meta content=1280 property=og:image:width><meta content=720 property=og:image:height><meta content=summary_large_image property=twitter:card><meta content=@learn_byexample property=twitter:site><link href="favicon.svg" rel=icon><link rel="shortcut icon"href="favicon.png"><link href="css/variables.css" rel=stylesheet><link href="css/general.css" rel=stylesheet><link href="css/chrome.css" rel=stylesheet><link href="FontAwesome/css/font-awesome.css" rel=stylesheet><link href="fonts/fonts.css" rel=stylesheet><link href="highlight.css" rel=stylesheet><link href="tomorrow-night.css" rel=stylesheet><link href="ayu-highlight.css" rel=stylesheet><link href="style.css" rel=stylesheet><body><script>var path_to_root = "";
            var default_theme = window.matchMedia("(prefers-color-scheme: dark)").matches ? "navy" : "light";</script><script>try {
                var theme = localStorage.getItem('mdbook-theme');
                var sidebar = localStorage.getItem('mdbook-sidebar');

                if (theme.startsWith('"') && theme.endsWith('"')) {
                    localStorage.setItem('mdbook-theme', theme.slice(1, theme.length - 1));
                }

                if (sidebar.startsWith('"') && sidebar.endsWith('"')) {
                    localStorage.setItem('mdbook-sidebar', sidebar.slice(1, sidebar.length - 1));
                }
            } catch (e) { }</script><script>var theme;
            try { theme = localStorage.getItem('mdbook-theme'); } catch(e) { }
            if (theme === null || theme === undefined) { theme = default_theme; }
            var html = document.querySelector('html');
            html.classList.remove('no-js')
            html.classList.remove('light')
            html.classList.add(theme);
            html.classList.add('js');</script><script>var html = document.querySelector('html');
            var sidebar = 'hidden';
            if (document.body.clientWidth >= 1080) {
                try { sidebar = localStorage.getItem('mdbook-sidebar'); } catch(e) { }
                sidebar = sidebar || 'visible';
            }
            html.classList.remove('sidebar-visible');
            html.classList.add("sidebar-" + sidebar);</script><nav aria-label="Table of contents"class=sidebar id=sidebar><div class=sidebar-scrollbox><ol class=chapter><li class="chapter-item expanded affix"><a href="cover.html">Cover</a><li class="chapter-item expanded affix"><a href="buy.html">Buy PDF/EPUB versions</a><li class="chapter-item expanded"><a href="preface.html"><strong aria-hidden=true>1.</strong> Preface</a><li class="chapter-item expanded"><a href="one-liner-introduction.html"><strong aria-hidden=true>2.</strong> One-liner introduction</a><li class="chapter-item expanded"><a href="line-processing.html"><strong aria-hidden=true>3.</strong> Line processing</a><li class="chapter-item expanded"><a href="in-place-file-editing.html"><strong aria-hidden=true>4.</strong> In-place file editing</a><li class="chapter-item expanded"><a href="field-separators.html"><strong aria-hidden=true>5.</strong> Field separators</a><li class="chapter-item expanded"><a href="record-separators.html"><strong aria-hidden=true>6.</strong> Record separators</a><li class="chapter-item expanded"><a href="using-modules.html"><strong aria-hidden=true>7.</strong> Using modules</a><li class="chapter-item expanded"><a href="multiple-file-input.html"><strong aria-hidden=true>8.</strong> Multiple file input</a><li class="chapter-item expanded"><a href="processing-multiple-records.html"><strong aria-hidden=true>9.</strong> Processing multiple records</a><li class="chapter-item expanded"><a href="two-file-processing.html"><strong aria-hidden=true>10.</strong> Two file processing</a><li class="chapter-item expanded"><a class=active href="dealing-with-duplicates.html"><strong aria-hidden=true>11.</strong> Dealing with duplicates</a><li class="chapter-item expanded"><a href="perl-rename-command.html"><strong aria-hidden=true>12.</strong> Perl rename command</a><li class="chapter-item expanded"><a href="Exercise_solutions.html"><strong aria-hidden=true>13.</strong> Exercise Solutions</a></li><br><hr><li class="chapter-item expanded"><i class="fa fa-github"id=git-repository-button></i><a href=https://github.com/learnbyexample/learn_perl_oneliners>   Source code</a><li class="chapter-item expanded"><i class="fa fa-home"id=home-button></i><a href="../index.html">   My Blog</a><li class="chapter-item expanded"><i class="fa fa-book"id=book-button></i><a href="../books.html">   My Books</a><li class="chapter-item expanded"><i class="fa fa-envelope"id=mail-button></i><a href=https://learnbyexample.gumroad.com/l/learnbyexample-weekly>   learnbyexample weekly</a><li class="chapter-item expanded"><i class="fa fa-twitter"id=twitter-button></i><a href=https://twitter.com/learn_byexample>   Twitter</a></ol></div><div class=sidebar-resize-handle id=sidebar-resize-handle></div></nav><div class=page-wrapper id=page-wrapper><div class=page><div id=menu-bar-hover-placeholder></div><div class="menu-bar sticky bordered"id=menu-bar><div class=left-buttons><button aria-label="Toggle Table of Contents"title="Toggle Table of Contents"aria-controls=sidebar class=icon-button id=sidebar-toggle type=button><i class="fa fa-bars"></i></button><button aria-label="Change theme"title="Change theme"aria-controls=theme-list aria-expanded=false aria-haspopup=true class=icon-button id=theme-toggle type=button><i class="fa fa-paint-brush"></i></button><ul aria-label=Themes class=theme-popup id=theme-list role=menu><li role=none><button class=theme id=light role=menuitem>Light (default)</button><li role=none><button class=theme id=rust role=menuitem>Rust</button><li role=none><button class=theme id=coal role=menuitem>Coal</button><li role=none><button class=theme id=navy role=menuitem>Navy</button><li role=none><button class=theme id=ayu role=menuitem>Ayu</button></ul><button aria-label="Toggle Searchbar"title="Search. (Shortkey: s)"aria-controls=searchbar aria-expanded=false aria-keyshortcuts=S class=icon-button id=search-toggle type=button><i class="fa fa-search"></i></button></div><h1 class=menu-title>Perl One-Liners Guide</h1><div class=right-buttons><a aria-label=Blog href="../index.html" title=Blog> <i class="fa fa-home"id=home-button></i> </a><a aria-label=Twitter href=https://twitter.com/learn_byexample title=Twitter> <i class="fa fa-twitter"id=twitter-button></i> </a><a aria-label="Git repository"title="Git repository"href=https://github.com/learnbyexample/learn_perl_oneliners> <i class="fa fa-github"id=git-repository-button></i> </a></div></div><div class=hidden id=search-wrapper><form class=searchbar-outer id=searchbar-outer><input placeholder="Search this book ..."aria-controls=searchresults-outer aria-describedby=searchresults-header id=searchbar name=searchbar type=search></form><div class="searchresults-outer hidden"id=searchresults-outer><div class=searchresults-header id=searchresults-header></div><ul id=searchresults></ul></div></div><script>document.getElementById('sidebar-toggle').setAttribute('aria-expanded', sidebar === 'visible');
                    document.getElementById('sidebar').setAttribute('aria-hidden', sidebar !== 'visible');
                    Array.from(document.querySelectorAll('#sidebar a')).forEach(function(link) {
                        link.setAttribute('tabIndex', sidebar === 'visible' ? 0 : -1);
                    });</script><div class=content id=content><main><div class=sidetoc><nav class=pagetoc></nav></div><h1 id=dealing-with-duplicates><a class=header href="dealing-with-duplicates.html#dealing-with-duplicates">Dealing with duplicates</a></h1><p>Often, you need to eliminate duplicates from an input file. This could be based on the entire line content or based on certain fields. These are typically solved with the <code>sort</code> and <code>uniq</code> commands. Advantages with Perl include regexp based field separator, record separator other than newline, input doesn't have to be sorted, and in general more flexibility because it is a programming language.<blockquote><p><img alt=info src="images/info.svg"> The <a href=https://github.com/learnbyexample/learn_perl_oneliners/tree/main/example_files>example_files</a> directory has all the files used in the examples.</blockquote><h2 id=whole-line-duplicates><a class=header href="dealing-with-duplicates.html#whole-line-duplicates">Whole line duplicates</a></h2><p>You can use the <code>uniq</code> function from the <code>List::Util</code> module or use a hash to retain only the first copy of duplicates from one or more input files.<pre><code class=language-bash>$ cat purchases.txt
coffee
tea
washing powder
coffee
toothpaste
tea
soap
tea

# same as: perl -MList::Util=uniq -e 'print uniq <>' purchases.txt
# can also use: perl -ne 'print if !exists $h{$_}; $h{$_}=1'
$ perl -ne 'print if !$h{$_}++' purchases.txt
coffee
tea
washing powder
toothpaste
soap
</code></pre><blockquote><p><img alt=info src="images/info.svg"> See also <a href=https://github.com/koraa/huniq>huniq</a>, a faster alternative for removing line based duplicates.</blockquote><h2 id=column-wise-duplicates><a class=header href="dealing-with-duplicates.html#column-wise-duplicates">Column wise duplicates</a></h2><p>The hash based solution is easy to adapt for removing field based duplicates. Just change <code>$_</code> to the required fields after setting the appropriate field separator.<pre><code class=language-bash>$ cat duplicates.txt
brown,toy,bread,42
dark red,ruby,rose,111
blue,ruby,water,333
dark red,sky,rose,555
yellow,toy,flower,333
white,sky,bread,111
light red,purse,rose,333

# based on the last field
# -l isn't needed if all the lines end with a newline character
$ perl -F, -ane 'print if !$h{$F[-1]}++' duplicates.txt
brown,toy,bread,42
dark red,ruby,rose,111
blue,ruby,water,333
dark red,sky,rose,555
</code></pre><p>Multiple fields example. As seen in the <a href="two-file-processing.html#comparing-fields">Comparing fields</a> section, you can either use comma separated values to construct the hash key or use a hash of hashes.<pre><code class=language-bash># based on the first and third fields
# can also use: perl -F, -ane 'print if !$h{$F[0]}{$F[2]}++'
$ perl -F, -ane 'print if !$h{$F[0],$F[2]}++' duplicates.txt
brown,toy,bread,42
dark red,ruby,rose,111
blue,ruby,water,333
yellow,toy,flower,333
white,sky,bread,111
light red,purse,rose,333
</code></pre><h2 id=duplicate-count><a class=header href="dealing-with-duplicates.html#duplicate-count">Duplicate count</a></h2><p>In this section, how many times a duplicate record is found plays a role in determining the output. First up, printing only a specific numbered duplicate.<pre><code class=language-bash># print only the second occurrence of duplicates based on the second field
$ perl -F, -ane 'print if ++$h{$F[1]} == 2' duplicates.txt
blue,ruby,water,333
yellow,toy,flower,333
white,sky,bread,111

# print only the third occurrence of duplicates based on the last field
$ perl -F, -ane 'print if ++$h{$F[-1]} == 3' duplicates.txt
light red,purse,rose,333
</code></pre><p>Next, printing only the last copy of duplicate. Since the count isn't known, the <code>tac</code> command comes in handy again.<pre><code class=language-bash># reverse the input line-wise, retain the first copy and then reverse again
$ tac duplicates.txt | perl -F, -ane 'print if !$h{$F[-1]}++' | tac
brown,toy,bread,42
dark red,sky,rose,555
white,sky,bread,111
light red,purse,rose,333
</code></pre><p>To get all the records based on a duplicate count, you can pass the input file twice. Then use the two file processing tricks to make decisions.<pre><code class=language-bash># all duplicates based on the last column
$ perl -F, -ane '!$#ARGV ? $h{$F[-1]}++ :
                 $h{$F[-1]}>1 && print' duplicates.txt duplicates.txt
dark red,ruby,rose,111
blue,ruby,water,333
yellow,toy,flower,333
white,sky,bread,111
light red,purse,rose,333

# all duplicates based on the last column, minimum 3 duplicates
$ perl -F, -ane '!$#ARGV ? $h{$F[-1]}++ :
                 $h{$F[-1]}>2 && print' duplicates.txt duplicates.txt
blue,ruby,water,333
yellow,toy,flower,333
light red,purse,rose,333

# only unique lines based on the third column
$ perl -F, -ane '!$#ARGV ? $h{$F[2]}++ :
                 $h{$F[2]}==1 && print' duplicates.txt duplicates.txt
blue,ruby,water,333
yellow,toy,flower,333
</code></pre><h2 id=summary><a class=header href="dealing-with-duplicates.html#summary">Summary</a></h2><p>This chapter showed how to work with duplicate contents for records and fields. If you don't need regexp based separators and if your input is too big to handle, then specialized command line tools like <code>sort</code> and <code>uniq</code> will be better suited.<h2 id=exercises><a class=header href="dealing-with-duplicates.html#exercises">Exercises</a></h2><blockquote><p><img alt=info src="images/info.svg"> The <a href=https://github.com/learnbyexample/learn_perl_oneliners/tree/main/exercises>exercises</a> directory has all the files used in this section.</blockquote><p><strong>1)</strong> Retain only the first copy of a line for the input file <code>lines.txt</code>. Case should be ignored while comparing the lines. For example, <code>hi there</code> and <code>HI TheRE</code> should be considered as duplicates.<pre><code class=language-bash>$ cat lines.txt
Go There
come on
go there
---
2 apples and 5 mangoes
come on!
---
2 Apples
COME ON

##### add your solution here
Go There
come on
---
2 apples and 5 mangoes
come on!
2 Apples
</code></pre><p><strong>2)</strong> Retain only the first copy of a line for the input file <code>twos.txt</code>. Assume space as the field separator with exactly two fields per line. Compare the lines irrespective of the order of the fields. For example, <code>hehe haha</code> and <code>haha hehe</code> should be considered as duplicates.<pre><code class=language-bash>$ cat twos.txt
hehe haha
door floor
haha hehe
6;8 3-4
true blue
hehe bebe
floor door
3-4 6;8
tru eblue
haha hehe

##### add your solution here
hehe haha
door floor
6;8 3-4
true blue
hehe bebe
tru eblue
</code></pre><p><strong>3)</strong> For the input file <code>twos.txt</code>, display only the unique lines. Assume space as the field separator with exactly two fields per line. Compare the lines irrespective of the order of the fields. For example, <code>hehe haha</code> and <code>haha hehe</code> should be considered as duplicates.<pre><code class=language-bash>##### add your solution here
true blue
hehe bebe
tru eblue
</code></pre></main><nav aria-label="Page navigation"class=nav-wrapper><a aria-label="Previous chapter"class="mobile-nav-chapters previous"title="Previous chapter"aria-keyshortcuts=Left href="two-file-processing.html" rel=prev> <i class="fa fa-angle-left"></i> </a><a aria-label="Next chapter"class="mobile-nav-chapters next"title="Next chapter"aria-keyshortcuts=Right href="perl-rename-command.html" rel=next> <i class="fa fa-angle-right"></i> </a><div style="clear: both"></div></nav></div></div><nav aria-label="Page navigation"class=nav-wide-wrapper><a aria-label="Previous chapter"class="nav-chapters previous"title="Previous chapter"aria-keyshortcuts=Left href="two-file-processing.html" rel=prev> <i class="fa fa-angle-left"></i> </a><a aria-label="Next chapter"class="nav-chapters next"title="Next chapter"aria-keyshortcuts=Right href="perl-rename-command.html" rel=next> <i class="fa fa-angle-right"></i> </a></nav></div><script>window.playground_copyable = true;</script><script charset=utf-8 src="elasticlunr.min.js"></script><script charset=utf-8 src="mark.min.js"></script><script charset=utf-8 src="searcher.js"></script><script charset=utf-8 src="clipboard.min.js"></script><script charset=utf-8 src="highlight.js"></script><script charset=utf-8 src="book.js"></script><script src="sidebar.js"></script>